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Overview 

This  presentation  is  based  on  some  insights 
gained  through  several  recent  “new  start” 
acquisitions.  While  the  focus  is  primarily  on  the 
pre-award  phase  of  the  acquisition,  a  few  post¬ 
award  issues  are  also  discussed,  and  a  few 
remedies  are  described  in  a  “Modest  Proposal” 
(with  apologies  to  Johnathan  Swift). 
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Pre-Award  Activities 

Request  for  Proposal  (RFP)  development 

•  “Do  it  faster” 

Source  Selection  /  Proposal  Evaluation 

•  Integrated  process  evaluation 
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RFP  Development, 

Does  this  look  familiar? 
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RFP  Development2 

How  ‘bout  this? 


Copyright  ©  2002  United  Feature  Syndicate,  Inc. 
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Do  It  Faster 

This  usually  gets  interpreted  as  “Make  it  Smaller”. 

•  Asking  for  less  detail 

•  “Intuiting”  processes  from  products 

•  Draconian  page  limits 

This  usually  doesn’t  work  too  well... 

•  Limits  the  ability  to  make  a  fully  informed  decision 

•  Generates  more  (and  more  rounds  of)  ENs 

•  May  actually  increase  the  effort  required  to  produce  and 
evaluate  proposals. 


I  would  have  written  a  shorter  letter ,  but  I  didn’t  have  the  time. 
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Current  Source  Selection/  Proposal 
Evaluation  Practice 
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Commonly  Used  “Integrated 
Processes”  Evaluation  Guidance 

Impose  the  SW-CMM®  Version  1 .1  “Level  3  Policy” 

•  Ask  for  copies  of  the  certificates 

•  Reduced  emphasis  on  SCESM  /  SDCE  with  on-site 
verification. 

“You  can’t  put  process  on  contract,  so  don’t  waste 
proposal  space  by  asking  about  it  directly.” 

•  Ask  for  the  OUTPUTS  of  their  processes  in  a  correctly 
formatted,  detailed  Integrated  Management  Plan  (IMP) 

•  Investigate  their  processes  during  an  “Executive  In- 
Plant  Review” 
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So,  What’s  Wrong  With  This?! 

“Impose  the  SW-CMM  ‘Level  3  Policy 

•  The  SW-CMM®  does  not  examine  all  of  the  process  areas 
necessary  for  program  success  (e.g.,  systems  engineering) 

•  The  SW-CMM®  evaluation  process  can  be  “gamed”  ...and  we 
have  provided  incentive  to  do  so 

•  Not  all  evaluation  methods  are  equal  (i.e.,  SCESM  vs.  CBA- 

|p|SM) 

•  Not  all  evaluators  are  equal  (e.g.,  John  Vu  (Boeing)  and  the 
“evaluator  credibility”  list) 

•  Evaluations  don’t  expire...  but  process  compliance  both 
evolves  and  devolves 


Level  3  is  a  good  start,  but  hardly  sufficient.  We  need  to 
have  a  more  sophisticated  understanding  of  a  contractor’s 
capabilities,  which  a  certificate  does  not  provide. 
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So,  What’s  Wrong  With  This?2 

“You  can’t  put  process  on  contract...” 

•  It’s  hard  to  both  write  and  evaluate  the  IMP 

-  Many  ways  to  reach  the  same  state,  some  more  efficient 
and  effective  than  others 

-  Entry,  exit  and  success  criteria  definition  necessary,  but 
not  usually  performed 

-  Linkages  frequently  not  identified 

-  “Can’t  see  the  forest  for  the  trees”  syndrome 

•  “Executive  reviews”  ineffective  at  determining  capability 

-  Reviewers  lack  background 

-  Reviewers  lack  access  at  appropriate  levels 

-  Reviews  are  too  short  for  proper  evaluation 


Don’t  eliminate  process  descriptions  or  capability  evaluations 
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Post  Award  Activities 

•  Risk  Management 

•  Integration  and  Testing  management  practices 
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Risk  Management 

Most  organizations  we  come  in  contact  with  have  defined  Risk 
Management  Processes. 

•  However,  their  cultures  vary  widely  on  their  ability  to  facilitate 
the  identification,  evaluation,  and  management  of  risk  in  their 
activities. 

You  need  to  evaluate  the  organizations  in  context  to  determine 
this.  Things  to  look  for: 

•  Large  number  of  risk  statements  =  a  lot  of  risk? 

•  How  are  risks  expressed? 

•  Who  (really)  identifies  and  qualifies  risks 

•  Other  names  for  risk  items  (watch  items,  issues,  etc.) 

•  Are  P0  and  Impact  assessments  reasonable,  re-evaluated? 


Distinguish  between  “risk  management”  and  “risk  avoidance/ 

denial.  ” 
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Integration  and  Test  -  PR  Management 

This  is  where  we  see  most  programs  really  going  wrong. 
Mostly,  it  is  because  this  is  where  problems  caused  by 
previous  activities  “come  home  to  roost”. 

Because  of  how  Problem  Reports  are  managed,  there  is 
no  way  of  telling  when  you  are  done. 

•  Level  of  Effort  activity 

•  “Capacity”  based  estimation 

•  Usually  no  estimate  of  effort  required  on  a  per-PR  basis 

•  Usually  no  detailed  tracking  of  effort  actuals 

•  Expected  effort  based  on  past  history  with  only  informal  / 
insufficient  data 

•  Confusion  between  “Severity”  and  “Priority” 

•  Scheduling  not  based  on  objective  data 

Start  defining  and  gathering  meaningful  metrics 
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Recent  Guidance 

Bottom  line:  Current  acquisition  practices/  trends  appear 
to  be  at  odds  with  recent  guidance  from  Dr.  Sambur 
(Assistant  Secretary  of  the  Air  Force  for  Acquisition), 
whose  Jan  6  2003  memo  states: 

“...Programs  must  elevate  these  (Systems  Engineering) 
disciplines  to  a  level  commensurate  with  other  programmatic 
considerations  such  as  cost  and  schedule.  ... 

“...I  am  further  directing  (PMs  and  the  ACE)  to  ensure  that 
disciplined  SE  practices  receive  adequate  consideration  in  all 
future  acquisitions,  whether  competitive  or  sole-source.  ... 

“...SAF/AQ  and  all  other  MDAs  will  not  sign  out  any  future 
ASPs  that  lack  the  necessary  and  sufficient  attention  to  SE.” 
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Alternatives  -  A  Modest  Proposal 

Question  assumptions  &  find  work-arounds 

•  You  can’t  put  process  on  contract 

-  You  can  put  process  compliance  and  quality  on 
contract 

-  Objective  and  subjective  measures  exist 

-  Incentives  based  on  award  fee  plan 

-  Lack  of  compliance  grounds  for  Termination  for 
Default 

Perform  your  own  software  capability  evaluation  as  part  of 
source  selection 

•  Details  of  the  method  do  not  matter  (SCE  vs.  SDCE) 

•  In  person,  in-plant  evaluations  are  critical 

•  Evaluate  all  software  development  locations 
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